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Summary 

A  training  need  exits  for  a  common,  cost-effective  virtual  reality  simulation  platform  for  aircraft 
maintenance  training.  The  need  is  driven  by  a  lack  of  concurrency  between  equipment  and  simulators, 
inaccessibility  of  equipment  and  simulators  for  training,  high  life-cycle  and  sustainment  costs  for  training 
simulators,  and  the  absence  of  full-scale  assets  or  mock-up  devices  at  training  squadrons.  In  the  absence  of 
training  assets  and  devices,  resident  training  has  been  restricted  to  traditional  methods  such  as  referencing 
technical  orders  and  academic  courseware.  An  additional  need  for  on-demand  training  at  operational  units 
has  contributed  to  increased  costs  by  allowing  maintainers  to  use  operational  flight  simulators  that  often 
require  declassification  for  maintenance  training  purposes.  This  paper  describes  the  assessment  of  the  Virtual 
Environment  Safe-for-maintenance  Trainer  and  development  of  the  Generalized  Operations  Simulation 
Environment,  the  next  generation  virtual  reality  aircraft  maintenance  training  program. 


1  Introduction 

The  Virtual  Environment  Safe-for-Maintenance  Trainer 

The  Virtual  Environment  Safe-for-Maintenance  Trainer  (VEST)  was  developed  in  1997  as  a  cooperative 
effort  among  the  363ld  Training  Squadron  (TRS),  Sheppard  Air  Force  Base  (AFB),  Texas,  the  Air  Force 
Research  Laboratory  (AFRL)  through  a  contract  with  Command  Technologies  Incorporated  (CTI)  at  Brooks 
Air  Force  Base,  Texas,  and  the  Air  Education  and  Training  Command  (AETC),  Randolph  AFB,  Texas.  The 
trainer  was  designed  to  meet  three  basic  needs:  (a)  train  switchology  for  F-15E-model  two  seat  cockpits,  (b) 
train  F-15E-model  weapons  stations  familiarization,  and  (c)  train  specific  ground  safe-for-maintenance  tasks 
on  the  F-15E  model.  Full-scale  F-15E  model  aircraft  are  not  available,  as  training  assets,  at  the  363rd  TRS. 

VEST  is  an  immersive  virtual  reality  (VR)  environment  that  provides  apprentice  technicians  demonstrations, 
drills,  and  checks  on  performance  maintaining  the  F-15E  model  aircraft.  The  value  of  VEST  is  its  low-cost 
replacement  of  the  actual  aircraft,  thus,  providing  apprentices  with  training  opportunities  they  would  not 
otherwise  have  prior  to  deployment.  VEST  provides  apprentices  with  contextualized,  three-dimensional, 
interactive  experiences  with  the  F-15E  model  front  and  rear  crew  stations,  weapons  stations,  and  ground 
safe-for-maintenance.  Apprentices  assigned  to  bases  with  F15-E  model  aircraft  are  required  to  complete  21 
lessons  in  VEST,  after  successfully  completing  baseline  training  on  the  F-15C  model  aircraft. 


Paper  presented  at  the  RTO  HFM  Symposium  on  “Advanced  Technologies  for  Military  Training”, 
held  in  Genoa,  Italy,  13  -  15  October  2003,  and  published  in  RTO-MP-HFM-101 . 
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There  are  two  VEST  stations  in  operation  at  the  363rd  TRS.  Apprentices  are  seated  and  interact  within  the 
virtual  environment  through  a  head  mounted  display  (HMD)  and  cabled  joystick  (see  Figure  1).  It  takes 
approximately  two  and  one  half  hours  to  complete  VEST.  Apprentices  are  given  two  attempts  to  achieve  a 
minimum  score  of  70%  to  pass  a  performance  check. 


HMD  cable 


Joystick 


Laser  beam  for  selecting  objects 
in  the  virtual  environment 


Representation  of  Joystick 
in  the  virtual  environment 


Figure  1.  VEST  system  in  operation 


The  Generalized  Operations  Simulation  Environment 

The  Generalized  Operations  Simulation  Environment  (GOSE)  project  is  a  collaborative  effort  between 
AETC  and  AFRL  to  develop  common,  cost-effective,  generalized  VR  training  platforms  for  aircraft 
maintenance  training,  undergraduate  pilot  training,  and  space  and  missile  training.  Our  initial  focus  is  the 
aircraft  maintenance  training  domain  since  it  provided  an  opportunity  to  build  on  the  VEST  architecture. 
Development  of  GOSE  involves  re-engineering  VEST  as  a  scalable,  modular,  immersive  VR  training  system 
comprised  of  PC-based  hardware  and  software.  GOSE  initiatives  include:  (a)  formalize  training  needs  across 
airframes,  (b)  design  instruction  to  meet  common  training  objectives,  (c)  create  a  common  training  platform 
that  supports  multiple  weapon  systems,  (d)  increase  personnel  and  equipment  safety;  (e)  increase  readiness  in 
technicians  trained  without  direct  hands-on  experience,  and  (f)  reduce  training  costs  using  “virtual” 
replacements  of  aircraft  assets  and  lower  life-cycle  and  sustainment  costs  of  VR  platforms. 

2  Evaluation  of  VEST 

The  363rd  TRS  is  continually  assessing  the  current  VEST  system.  Results  from  the  assessment  of  VEST  have 
helped  us  identify  needed  improvements  to  the  GOSE  architecture.  The  ongoing  assessment  of  VEST 
encompasses  apprentice  performance  scores  in  the  VR  environment  and  their  reactions  to  the  training 
experience.  Apprentice  technicians  are  asked  to  provide  comments  about  their  experience,  strengths  and 
weaknesses  of  the  system,  and  any  adverse  physical  effects  encountered. 

Performance  data  and  comments  collected  from  84  apprentices  assigned  to  complete  VEST  are  reported  here. 
Two  apprentices  were  unable  to  complete  VEST  and  their  data  was  excluded  from  the  analyses.  The  average 
performance  scores  across  the  VEST  drills  by  number  of  trials  is  depicted  in  Table  1.  The  system  failed  to 
capture  “aircraft  safe-for-maintenance  drill”  scores  for  three  apprentices  so  their  overall  performance  scores 
could  not  be  computed.  The  overall  average  performance  score  was  80.3%.  Six  of  the  82  apprentices  had 
overall  average  scores  below  70%.  Performance  results  indicated  apprentices  had  the  greatest  difficulty  with 
the  “left  console  aft  cockpit  switch  drill.”  Even  after  the  maximum  number  of  trials  (two)  only  seven 
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apprentices  were  able  to  pass  with  acceptable  scores  (100%).  “Review  aft  cockpit  drill”  and  “weapons  station 
drill”  also  indicated  some  degree  of  difficulty  for  19  and  56  of  the  82  apprentices,  respectively.  After  two 
tries  these  apprentices  scored  only  slightly  over  the  70%  pass  rate.  Practice  had  the  largest  impact  on  the 
“aircraft  safe-for-maintenance  drill”  where  those  who  required  a  second  try  at  attaining  a  passing  score 
outscored  those  who  passed  on  a  single  try  (t  (75)  =  3.34;  pvalue  =  .001). 


Table  1.  Average  perfonnance  of  scores  on  VEST  drills  by  number  of  tries  (n=number  of  apprentices) 


Number  of  Tries 

Number  of  Tries 

Drill 

1 

2 

Drill 

1 

2 

Left  console  forward  cockpit  switch  drill 

81.6 

(n=53) 

81.1 

(n=29) 

Left  console  aft  cockpit  switch  drill 

— 

25.6 

(n=82) 

Main  console  forward  cockpit  switch  drill 

82.0 

(n=59) 

82.1 

(n=23) 

Main  console  aft  cockpit  switch  drill 

91.4 

(n=77) 

90.0 

(n=5) 

Right  console  forward  cockpit 

90.0 

(n=77) 

82.6 

(n=5) 

Right  console  aft  cockpit  switch  drill 

84.4 

(n=73) 

81.2 

(n=9) 

Review  aft  cockpit 

84.4 

(n=63) 

74.2 

(n=19) 

Weapons  station  drill 

99.9 

(n=26) 

77.7 

(n=56) 

Safety  device  drill 

87.1 

(n=73) 

89.0 

(n=9) 

Aircraft  safe-for-maintenance  intro  drill 

91.6 

(n=79) 

95.5 

(n=2) 

Aircraft  safe-for-maintenance  drill 

76.4 

(n=22) 

84.8 

(n=55) 

Apprentice  comments  were  assigned  a  category  and  a  valance  (positive,  negative,  or  neutral)  by  two  raters. 
Fourteen  coding  categories  were  used  that  encompassed  all  263  comments  (Table  2).  Cohen’s  index  “kappa” 
was  used  to  derive  inter-rater  reliabilities,  which  were  .95  for  category  assignment  and  .90  for  valance 
assignment.  Raters  came  to  an  agreement  on  divergent  category  and  valance  assignments,  after  making  initial 
ratings,  before  computing  the  number  of  comments  within  categories. 


Table  2.  Category  Definitions 


Category 

Definition 

Aircraft: 

comparison  between  VEST  and  F-15E 

Content: 

reference  to  instructional  material 

Experience: 

reaction  to  VEST 

Fix: 

suggestion  to  improve  VEST 

HMD: 

reference  to  VR  HMD 

Learn: 

testimony  to  learning  something 

Learning  Style: 

preference  for  an  instructional  method 

Media: 

reference  to  training  tool 

Physiology: 

physical  reaction  to  VEST 

Pointer: 

reference  to  interface  (hardware/software)  for  object  selection 
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Unreliability: 

reference  to  system  glitches 

Usability: 

reference  to  interactions  with  VEST  instructional  method 

Visual: 

reference  to  quality  of  visual  experience/graphical  representation  of  aircraft 

Voice: 

reference  to  quality  of  aural  experience 

Table  3  presents  the  frequencies  of  comments  by  category  and  valance.  On  average,  apprentices  made  3.1 
comments  (range  1-7).  The  average  number  of  positive  comments  was  1.3  (range  0-4).  The  average  number 
of  negative  comments  was  1.4  (range  0-5).  The  average  number  of  neutral  comments  was  .4  (range  0-2).  The 
comment  distributions  were  all  positively  skewed,  showing  fewer  people  made  increasing  numbers  of 
comments.  Categories  are  divided  in  the  table  between  mostly  positive,  mostly  neutral,  and  mostly  negative 
comments.  Overall,  41.5%  of  comments  were  positive,  13.3%  were  neutral,  and  45.2%  were  negative.  The 
division  of  comment  categories  by  valance  reveals  107  positive  comments  distributed  over  5  distinct 
categories  and  106  negative  comments  distributed  over  8  distinct  categories.  All  comments  (23)  in  the  “Fix” 
category,  except  one,  were  assigned  a  neutral  valance  table. 


Table  3.  Number  of  Apprentice  Comments  by  Category  and  Valance 


Mostly  Positive 

Mostly  Negative 

Category 

Positive 

Negative 

Neutral 

Category 

Positive 

Negative 

Neutral 

Experience 

39 

8 

8 

HMD 

— 

5 

— 

Learn 

37 

— 

1 

Learning  Style 

— 

12 

1 

Media 

18 

2 

1 

Physiology 

— 

20 

— 

Aircraft 

8 

— 

— 

Pointer 

— 

27 

— 

Constant 

5 

3 

1 

Unreliability 

— 

9 

— 

COUNT 

107 

13 

11 

Usability 

2 

21 

— 

Mostly  Neutral 

Visual 

— 

10 

— 

Voice 

— 

3 

— 

Category 

Positive 

Negative 

Neutral 

COUNT 

2 

106 

1 

Fix 

— 

1 

23 

No  negative  comments  were  made  about  learning  or  the  value  of  gaining  experience  with  an  E-model 
aircraft.  Example  positive  comments  are  “The  VR  program  served  its  purpose  very  well  and  was  extremely 
educating”  and  “gives  you  a  realistic  view  of  the  aircraft  and  the  loading  stations.”  No  positive  comments 
were  made  about  the  peripheral  devices,  graphical  representations,  voice  synthesis,  physiology  response, 
reliability,  and  learning  styles. 

A  median  split  was  conducted  to  create  two  groups  of  apprentices — those  who  performed  at  and  below  50% 
of  the  class  (average  overall  score  of  80.5%)  and  those  who  performed  above  50%  of  the  class.  Chi-Square 
tests  revealed  no  difference  in  the  types  of  comments  apprentices  made  whether  they  performed  at  or  above 
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the  median  score  or  below  the  median  score.  Nor  were  differences  found  in  the  number  of  positive  and 
negative  comments  across  categories  based  on  the  median  split.  No  relationship  was  found  between  overall 
performance  scores  and  number  of  comments  made  (r  =  .099).  Apprentices  were  divided  into  three  groups 
based  on  a  comparison  of  the  number  of  positive  versus  negative  comments  (+>-;  +  =  -;  +  <  -).  No 
significant  differences  were  found  in  performance  scores  across  the  three  groups  [M+>.  =  81.3  (n  =  30); 

M+  =  .  =  80.1  (n  =  17);  M+<-  =  79.5  (n  =  32)].  (Wenzel,  Castillo,  &  Baker  2002). 

3  GOSE  Development  Methods  and  Tools 

Technical  specifications  for  VEST  are  presented  in  Table  4.  A  baseline  configuration  for  VEST  was 
completed  before  start  of  the  GOSE  migration.  The  baseline  configuration  included:  video  capture,  data 
backup,  tutorial  evaluation  and  demonstration. 


Table  4.  VEST  technical  specifications. 


Hardware 

Software 

Instructor 

Workstation 

(1) 

Silicon  Graphics  02  Workstation 

180MHZ  R5000SC  (secondary  cache) 

•  128MB  RAM,  4  GB  Hard  drive 

•  Mouse,  keyboard,  monitor 

•  10  Base-T  to  BNC  multi-port  repeater 

•  DAT  tape  SCSI  Backup  System 

•  10  DAT  tapes  (4mm  DAT-4-8  GB) 

•  Network  File  System  (NFS) 

•  IRIX  version:  6.3  operating  system 

•  C++  compiler  for  Irix  6.3 

•  VEGA™  SP  Development  System 

•  Ez3D  Modeler 

Student 

Workstations 

(2) 

Silicon  Graphics  Indigo2  Maximum  IMPACT  (Indigo2) 
workstation,  195HZ  R10000 

•  128MB  RAM,  4  MB  texture  memory 

•  4  GB  hard  drive 

•  6  DOF  Motion  tracking  device 

•  2  receivers  with  cable 

•  2  flock  of  birds  electronics  units 

•  1  flock  transmitter  with  10  foot  cable 

•  HMD  system:  Virtual  Research  V6  with  control  box 

•  Two  Flock-of-birds  model  6  degrees  of  freedom 

•  Cyberwand,  serial  version 

•  IRIX  version:  6.2  operating  system 

•  VIVIDS  0.82b  for  the  SGI™ 

•  Truetalk™  Text-to-Speech  engine 

•  Vega™  3.0  SP  Runtime 

•  Performer  2.0.4 

Table  5  contains  technical  specifications  for  GOSE.  GOSE  is  being  developed  as  a  PC-based  platform 
enhanced  with  data  gloves,  HMD  and  tracking  system  that  will  provide  end-users  six  degrees  of  freedom  in 
movement.  These  specifications  in  part  address  issues  raised  in  the  evaluation  of  VEST.  The  enhancements 
are  expected  to  improve  navigation  within  the  virtual  environment  (see  figure  2). 
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Table  5.  GOSE  technical  specifications. 


Hardware 

Software 

Instructor 

Workstation 

(1) 

IBM  Z  Pro,  2  GHZ  (2  processors) 

•  2  GB  RAM,  GeForce4™  video  card 

•  Mouse,  keyboard,  monitor 

•  XP  operating  system 

•  Vega™  Prime 

Student 

Workstation 

(1) 

IBM  Z  Pro,  2  GHZ  (2  processors) 

•  2  GB  RAM,  GeForce4™  video  card 

•  Virtual  Research  VS  HMD 

•  Two  Flock-of-birds 

•  Magnetic  tracking  system  from  Ascension 

•  Pinch  Gloves 

•  Phantom™  haptic  device 

•  CrystalEyes  stereographic  glasses 

•  XP  operating  system 

•  Vega™  Prime 

•  GHOST™  software  development  kit 

Figure  2.  Concept  for  GOSE  immersive  system  with  improved  HMD  and  data  gloves. 


Inclusion  of  a  haptic  (feedback)  interface  and  stereoscopic  glasses,  along  with  use  of  three-dimensional 
models  and  synthetic  voice  commands,  in  the  next  phases  of  development  should  provide  the  end-user 
greater  flexibility  and  increased  realism  in  the  immersive  environment.  The  improvements  are  expected  to 
further  increase  training  effectiveness  and  reduce  cybersickness.  Evaluation  of  GOSE  is  planned  following 
completion  of  the  each  phase  of  development.  For  comparative  purposes,  assessment  criteria  used  to  evaluate 
GOSE  will  include  the  same  criteria  used  to  evaluate  VEST. 

4  Discussion 

The  VEST  evaluation  results  indicate  graphical  representations,  joystick  pointer,  and  cybersickness  are  end- 
user  concerns.  The  design  of  GOSE  is  intended  to  address  these  issues  and  heightened  usability  for  the  end- 
user. 

Commercial-off-the-shelf  (COTS)  applications  to  be  used  in  GOSE  accept  embedded  graphics 
(photographs),  rather  than  constructed  graphics  in  creating  the  virtual  airframe.  Reductions  in  motion  lag 
time,  making  the  airframe  “solid,”  and  surround  detail  are  needed  enhancements  to  sensory  realism.  It  is 
likely  that  such  changes  in  GOSE  will  reduce  feelings  of  nausea  and  dizziness.  The  multimedia  resource 
elements  of  VR  (diagrams,  images,  text,  video,  etc.)  add  richness  and  depth  to  the  learning  experience  and 
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can  be  used  to  facilitate  strong  cognitive  links  (Hoffman  &  Murray,  1999).  For  example,  training  on  part 
replacement  inside  an  airplane  fuselage  or  an  automobile  engine  can  be  clarified  by  making  the  virtual 
machine’s  outer  layers  invisible  or  transparent. 


Peripheral  devices,  e.g.,  Pinch  Gloves™  are  available  to  allow  users  to  interact  with  the  virtual  airframe  in 
manners  similar  to  how  technicians  interact  with  an  actual  airframe.  The  VEST  joystick  will  be  replaced  in 
GOSE  with  data  gloves  to  enable  the  experience  to  more  closely  resemble  hands-on  training.  Apprentices 
will  be  required  to  reach  out  to  interact  with  switches  and  ground  maintenance  safety  pins.  Apprentices  can 
be  seated  when  working  in  the  cockpit  and  agile  when  safe-ing  the  aircraft  for  maintenance. 

In  the  VEST  trainer,  apprentices  were  encouraged  to  take  breaks  every  15  to  30  minutes  and  not  to  stay  in 
longer  than  45  minutes;  however,  some  chose  to  complete  the  exercises  without  breaks.  Reported  adverse 
physical  reactions  to  the  VR  experience  such  as  headache  and  eyestrain,  may  be  due  to  the  length  of 
immersion  (Gupta,  Klein,  &  Wantland,  1996).  The  length  of  immersion  may  also  affect  the  instructional 
effectiveness  of  GOSE.  There  are  reasonable  points  in  training  content  for  forced  breaks  that  would  shorten 
immersion  time  and  support  learning.  In  response  to  end-users  reactions,  forced-breaks  in  the  GOSE  training 
are:  (a)  cockpit  switch  familiarization,  (b)  weapons  station  and  safety  device  identification,  and  (c)  ground 
safe-for-maintenance  procedures. 

VR  adds  to  learning  through  experience.  End-users  learn  “how  to”  and  perform  physical  tasks  in  real-time  in 
the  virtual  environment  without  risk  to  an  apprentice’s  safety  (Eline,  1998).  However,  instructional 
improvements  are  needed  in  GOSE  to  (a)  control  for  information  overload,  (b)  expand  content  areas,  (c) 
extend  opportunities  to  practice,  (d)  guide  learning,  and  (e)  increase  accessibility  to  the  trainer.  The  addition 
of  working  collaboratively  in  the  virtual  environment  would  further  enhance  the  learning  experience  and 
training  capability  of  GOSE. 

5  Conclusion 

Results  from  the  VEST  assessment  help  lead  the  way  to  GOSE,  beginning  with  addressing  realism  and 
usability  issues  and  guiding  necessary  improvements  to  the  instructional  design.  GOSE  will  provide  a  VR 
platform  to  continue  the  research  to  better  understand  the  physiological  responses  (e.g.,  headache,  eyestrain, 
disorientation,  nausea,  muscle  stress)  to  the  virtual  environment  and  to  continue  to  explore  cutting-edge 
methods  and  tools  to  increase  training  effectiveness  and  training  transfer.  Assessment  of  GOSE  should 
include  data  to  determine  the  extent  to  which  training  with  VR  systems  transfers  to  the  operational 
environment.  Elements  of  the  operational  environment  such  as  visual  and  auditory  representations  of  the 
flight  line  incorporated  in  the  virtual  environment  would  likely  enhance  transfer  of  training. 
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